Theory Lunch

Towards a Systematic Understanding of Exempt and Non-Exempt Biases in Algorithms

Sanghamitra Dutta

March 10, 2021 (Zoom - See email or contact organizers for link)

Talk Video on YouTube

Abstract: How do we enable machine learning models to not only be ethical but also comply with the law? This is a key question today as machine learning begins to affect all aspects of our lives, including high-stakes applications such as hiring, education, etc., and yet, has surprisingly received less attention. A good starting point is Title VII of the US Civil Rights Act, which prohibits employment discrimination based on race, color, religion, gender, etc. Included in Title VII is a subtle and important aspect that has implications for the machine learning models being used today: Biases that can be explained by a business necessity are exempt. E.g., bias arising due to code-writing skills may be deemed exempt for a software engineering job if considered a business necessity, while bias due to an aptitude test may not be (e.g. Griggs v. Duke Power ‘71). This leads to a pressing question bridging both fairness and explainability: How to quantify (and, if needed, remove) non-exempt bias, i.e., illegal bias which cannot be explained by business necessities. To arrive at a measure for non-exempt bias, I adopt a rigorous axiomatic approach that brings together concepts in information theory, in particular, an emerging body of work called Partial Information Decomposition, with Simpson's paradox and causal inference. I first examine several toy examples (thought experiments) that help us arrive at desirable properties (axioms) for a measure of non-exempt bias, and then, propose a counterfactual measure that satisfies these properties. I also obtain an impossibility result showing that no observational measure of non-exempt bias can satisfy all of the desired properties, which leads us to relax our goals and examine alternative observational measures that satisfy only some of these properties for purely observational applications. Finally, I discuss case studies to demonstrate how we can train models that selectively remove non-exempt bias, also providing a perspective into accuracy-bias trade-offs.